Ensemble is a general way of improving the accuracy and stability of learning models, especially for the generalization ability on small datasets. Compared with tree-based methods, relatively less works have been devoted to an in-depth study on effective ensemble design for neural networks. In this paper, we propose a principled ensemble technique by constructing the so-called diversified ensemble layer to combine multiple networks as individual modules. We theoretically show that each individual model in our ensemble layer corresponds to weights in the ensemble layer optimized in different directions. Meanwhile, the devised ensemble layer can be readily integrated into popular neural architectures, including CNNs, RNNs, and GCNs. Extensive experiments are conducted on public tabular datasets, images, and texts. By adopting weight sharing approach, the results show our method can notably improve the accuracy and stability of the original neural networks with ignorable extra time and space overhead.

In other words, I am wondering how we can make sure the improvements presented by using DEns-NN contributes to generalization? One possible suggestion could be tracking L_d (diversity loss function) during training on both training data and some validation data; it can be insightful to see how much of error reduction is correlated/due to L_d. 3- In the same line, I am also wondering how the hyperparameters in R-Forest, XGBoost, NN are set? Are they different across different datasets? Are they set individually for each dataset via a validation set? 4- The paper is written clearly; however, it can improve by revisiting the use of notations; for example N is used for different purposes. Also, there are some typos.

Two referees support accept, one weak-accept and one indicates reject. Rebuttal clarified R3's concerns regarding two related work directions, but R3 did not respond during the reviewer discussion. R4 (weak-accept) did engage and was happy with the rebuttal connected with their concerns. R1 and R2 were happier with their accept stance post-rebuttal. I therefore think that this paper should be accepted.

Ensemble is a general way of improving the accuracy and stability of learning models, especially for the generalization ability on small datasets. Compared with tree-based methods, relatively less works have been devoted to an in-depth study on effective ensemble design for neural networks. In this paper, we propose a principled ensemble technique by constructing the so-called diversified ensemble layer to combine multiple networks as individual modules. We theoretically show that each individual model in our ensemble layer corresponds to weights in the ensemble layer optimized in different directions. Meanwhile, the devised ensemble layer can be readily integrated into popular neural architectures, including CNNs, RNNs, and GCNs.